Multi-objective evolution for Generalizable Policy Gradient Algorithms

Performance, generalizability, and stability are three Reinforcement Learning (RL) challenges relevant to many practical applications in which they present themselves in combination. Still, state-of-the-art RL algorithms fall short when addressing multiple RL objectives simultaneously and current human-driven design practices might not be well-suited for multi-objective RL. In this paper we present MetaPG, an evolutionary method that discovers new RL algorithms represented as graphs, following a multi-objective search criteria in which different RL objectives are encoded in separate fitness scores. Our findings show that, when using a graph-based implementation of Soft Actor-Critic (SAC) to initialize the population, our method is able to find new algorithms that improve upon SAC's performance and generalizability by 3% and 17%, respectively, and reduce instability up to 65%. In addition, we analyze the graph structure of the best algorithms in the population and offer an interpretation of specific elements that help trading performance for generalizability and vice versa. We validate our findings in three different continuous control tasks: RWRL Cartpole, RWRL Walker, and Gym Pendulum.Comment: 23 pages, 12 figures, 10 table

Understanding HTML with Large Language Models

Large language models (LLMs) have shown exceptional performance on a variety of natural language tasks. Yet, their capabilities for HTML understanding -- i.e., parsing the raw HTML of a webpage, with applications to automation of web-based tasks, crawling, and browser-assisted retrieval -- have not been fully explored. We contribute HTML understanding models (fine-tuned LLMs) and an in-depth analysis of their capabilities under three tasks: (i) Semantic Classification of HTML elements, (ii) Description Generation for HTML inputs, and (iii) Autonomous Web Navigation of HTML pages. While previous work has developed dedicated architectures and training procedures for HTML understanding, we show that LLMs pretrained on standard natural language corpora transfer remarkably well to HTML understanding tasks. For instance, fine-tuned LLMs are 12% more accurate at semantic classification compared to models trained exclusively on the task dataset. Moreover, when fine-tuned on data from the MiniWoB benchmark, LLMs successfully complete 50% more tasks using 192x less data compared to the previous best supervised model. Out of the LLMs we evaluate, we show evidence that T5-based models are ideal due to their bidirectional encoder-decoder architecture. To promote further research on LLMs for HTML understanding, we create and open-source a large-scale HTML dataset distilled and auto-labeled from CommonCrawl

Evolving Reinforcement Learning Algorithms

We propose a method for meta-learning reinforcement learning algorithms by searching over the space of computational graphs which compute the loss function for a value-based model-free RL agent to optimize. The learned algorithms are domain-agnostic and can generalize to new environments not seen during training. Our method can both learn from scratch and bootstrap off known existing algorithms, like DQN, enabling interpretable modifications which improve performance. Learning from scratch on simple classical control and gridworld tasks, our method rediscovers the temporal-difference (TD) algorithm. Bootstrapped from DQN, we highlight two learned algorithms which obtain good generalization performance over other classical control tasks, gridworld type tasks, and Atari games. The analysis of the learned algorithm behavior shows resemblance to recently proposed RL algorithms that address overestimation in value-based methods.Comment: ICLR 2021 Oral. See project website at https://sites.google.com/view/evolvingr

Leveraging 16S rRNA Microbiome Sequencing Data to Identify Bacterial Signatures for Irritable Bowel Syndrome

Irritable bowel syndrome (IBS) is a chronic gastrointestinal disorder characterized by abdominal pain or discomfort. Previous studies have illustrated that the gut microbiota might play a critical role in IBS, but the conclusions of these studies, based on various methods, were almost impossible to compare, and reproducible microorganism signatures were still in question. To cope with this problem, previously published 16S rRNA gene sequencing data from 439 fecal samples, including 253 IBS samples and 186 control samples, were collected and processed with a uniform bioinformatic pipeline. Although we found no significant differences in community structures between IBS and healthy controls at the amplicon sequence variants (ASV) level, machine learning (ML) approaches enabled us to discriminate IBS from healthy controls at genus level. Linear discriminant analysis effect size (LEfSe) analysis was subsequently used to seek out 97 biomarkers across all studies. Then, we quantified the standardized mean difference (SMDs) for all significant genera identified by LEfSe and ML approaches. Pooled results showed that the SMDs of nine genera had statistical significance, in which the abundance of Lachnoclostridium, Dorea, Erysipelatoclostridium, Prevotella 9, and Clostridium sensu stricto 1 in IBS were higher, while the dominant abundance genera of healthy controls were Ruminococcaceae UCG-005, Holdemanella, Coprococcus 2, and Eubacterium coprostanoligenes group. In summary, based on six published studies, this study identified nine new microbiome biomarkers of IBS, which might be a basis for understanding the key gut microbes associated with IBS, and could be used as potential targets for microbiome-based diagnostics and therapeutics

Epidemiological characteristics, clinical presentations, and prognoses of pediatric brain tumors: Experiences of national center for children’s health

BackgroundWe aimed to describe the epidemiological characteristics, clinical presentations, and prognoses in a national health center for children.MethodsFrom January 2015 to December 2020, 484 patients aged 0-16 years, who were diagnosed with brain tumors and received neurosurgery treatment, were enrolled in the study. Pathology was based on the World Health Organization 2021 nervous system tumor classification, and tumor behaviors were classified according to the International Classification of Diseases for Oncology, third edition.ResultsAmong the 484 patients with brain tumors, the median age at diagnosis was 4.62 [2.19, 8.17] years (benign tumors 4.07 [1.64, 7.13] vs. malignant tumors 5.36 [2.78, 8.84], p=0.008). The overall male-to-female ratio was 1.33:1(benign 1.09:1 vs. malignant 1.62:1, p=0.029). Nausea, vomiting, and headache were the most frequent initial symptoms. The three most frequent tumor types were embryonal tumors (ET, 22.8%), circumscribed astrocytic gliomas (20.0%), and pediatric-type diffuse gliomas (11.0%). The most common tumor locations were the cerebellum and fourth ventricle (38.67%), the sellar region (22.9%) and ventricles (10.6%). Males took up a higher proportion than females in choroid plexus tumors (63.6%), ET (61.1%), ependymal tumors (68.6%), and germ cell tumors (GCTs, 78.1%). Patients were followed for 1 to 82 months. The overall 5-year survival rate was 77.5%, with survival rates of 91.0% for benign tumors and 64.6% for malignant tumors.ConclusionBrain tumors presented particularly sex-, age-, and regional-dependent epidemiological characteristics. Our results were consistent with previous reports and might reflect the real epidemiological status in China

Comparative transcriptome sequencing of germline and somatic tissues of the Ascaris suum gonad

<p>Abstract</p> <p>Background</p> <p><it>Ascaris suum </it>(large roundworm of pigs) is a parasitic nematode that causes substantial losses to the meat industry. This nematode is suitable for biochemical studies because, unlike <it>C. elegans</it>, homogeneous tissue samples can be obtained by dissection. It has large sperm, produced in great numbers that permit biochemical studies of sperm motility. Widespread study of <it>A. suum </it>would be facilitated by more comprehensive genome resources and, to this end, we have produced a gonad transcriptome of <it>A. suum</it>.</p> <p>Results</p> <p>Two 454 pyrosequencing runs generated 572,982 and 588,651 reads for germline (TES) and somatic (VAS) tissues of the <it>A. suum </it>gonad, respectively. 86% of the high-quality (HQ) reads were assembled into 9,955 contigs and 69,791 HQ reads remained as singletons. 2.4 million bp of unique sequences were obtained with a coverage that reached 16.1-fold. 4,877 contigs and 14,339 singletons were annotated according to the <it>C. elegans </it>protein and the Kyoto Encyclopedia of Genes and Genomes (KEGG) protein databases. Comparison of TES and VAS transcriptomes demonstrated that genes participating in DNA replication, RNA transcription and ubiquitin-proteasome pathways are expressed at significantly higher levels in TES tissues than in VAS tissues. Comparison of the <it>A. suum </it>TES transcriptome with the <it>C. elegans </it>microarray dataset identified 165 <it>A. suum </it>germline-enriched genes (83% are spermatogenesis-enriched). Many of these genes encode serine/threonine kinases and phosphatases (KPs) as well as tyrosine KPs. Immunoblot analysis further suggested a critical role of phosphorylation in both testis development and spermatogenesis. A total of 2,681 <it>A. suum </it>genes were identified to have associated RNAi phenotypes in <it>C. elegans</it>, the majority of which display embryonic lethality, slow growth, larval arrest or sterility.</p> <p>Conclusions</p> <p>Using deep sequencing technology, this study has produced a gonad transcriptome of <it>A. suum</it>. By comparison with <it>C. elegans </it>datasets, we identified sets of genes associated with spermatogenesis and gonad development in <it>A. suum</it>. The newly identified genes encoding KPs may help determine signaling pathways that operate during spermatogenesis. A large portion of <it>A. suum </it>gonadal genes have related RNAi phenotypes in <it>C. elegans </it>and, thus, might be RNAi targets for parasite control.</p

Validation of the ITS2 Region as a Novel DNA Barcode for Identifying Medicinal Plant Species

BACKGROUND: The plant working group of the Consortium for the Barcode of Life recommended the two-locus combination of rbcL+matK as the plant barcode, yet the combination was shown to successfully discriminate among 907 samples from 550 species at the species level with a probability of 72%. The group admits that the two-locus barcode is far from perfect due to the low identification rate, and the search is not over. METHODOLOGY/PRINCIPAL FINDINGS: Here, we compared seven candidate DNA barcodes (psbA-trnH, matK, rbcL, rpoC1, ycf5, ITS2, and ITS) from medicinal plant species. Our ranking criteria included PCR amplification efficiency, differential intra- and inter-specific divergences, and the DNA barcoding gap. Our data suggest that the second internal transcribed spacer (ITS2) of nuclear ribosomal DNA represents the most suitable region for DNA barcoding applications. Furthermore, we tested the discrimination ability of ITS2 in more than 6600 plant samples belonging to 4800 species from 753 distinct genera and found that the rate of successful identification with the ITS2 was 92.7% at the species level. CONCLUSIONS: The ITS2 region can be potentially used as a standard DNA barcode to identify medicinal plants and their closely related species. We also propose that ITS2 can serve as a novel universal barcode for the identification of a broader range of plant taxa

Guidelines for the use and interpretation of assays for monitoring autophagy (3rd edition)

In 2008 we published the first set of guidelines for standardizing research in autophagy. Since then, research on this topic has continued to accelerate, and many new scientists have entered the field. Our knowledge base and relevant new technologies have also been expanding. Accordingly, it is important to update these guidelines for monitoring autophagy in different organisms. Various reviews have described the range of assays that have been used for this purpose. Nevertheless, there continues to be confusion regarding acceptable methods to measure autophagy, especially in multicellular eukaryotes. For example, a key point that needs to be emphasized is that there is a difference between measurements that monitor the numbers or volume of autophagic elements (e.g., autophagosomes or autolysosomes) at any stage of the autophagic process versus those that measure fl ux through the autophagy pathway (i.e., the complete process including the amount and rate of cargo sequestered and degraded). In particular, a block in macroautophagy that results in autophagosome accumulation must be differentiated from stimuli that increase autophagic activity, defi ned as increased autophagy induction coupled with increased delivery to, and degradation within, lysosomes (inmost higher eukaryotes and some protists such as Dictyostelium ) or the vacuole (in plants and fungi). In other words, it is especially important that investigators new to the fi eld understand that the appearance of more autophagosomes does not necessarily equate with more autophagy. In fact, in many cases, autophagosomes accumulate because of a block in trafficking to lysosomes without a concomitant change in autophagosome biogenesis, whereas an increase in autolysosomes may reflect a reduction in degradative activity. It is worth emphasizing here that lysosomal digestion is a stage of autophagy and evaluating its competence is a crucial part of the evaluation of autophagic flux, or complete autophagy. Here, we present a set of guidelines for the selection and interpretation of methods for use by investigators who aim to examine macroautophagy and related processes, as well as for reviewers who need to provide realistic and reasonable critiques of papers that are focused on these processes. These guidelines are not meant to be a formulaic set of rules, because the appropriate assays depend in part on the question being asked and the system being used. In addition, we emphasize that no individual assay is guaranteed to be the most appropriate one in every situation, and we strongly recommend the use of multiple assays to monitor autophagy. Along these lines, because of the potential for pleiotropic effects due to blocking autophagy through genetic manipulation it is imperative to delete or knock down more than one autophagy-related gene. In addition, some individual Atg proteins, or groups of proteins, are involved in other cellular pathways so not all Atg proteins can be used as a specific marker for an autophagic process. In these guidelines, we consider these various methods of assessing autophagy and what information can, or cannot, be obtained from them. Finally, by discussing the merits and limits of particular autophagy assays, we hope to encourage technical innovation in the field